THE CONJUGACY PROBLEM IN AUTOMATON GROUPS 

IS NOT SOLVABLE 



ZORAN SUNIC AND ENRIC VENTURA 



Abstract. (Free-abelian)-by-free, self-similar groups generated by finite self- 
similar sets of tree automorphisms and having unsolvable conjugacy problem 
are constructed. Along the way, finitely generated, orbit undecidable, free 
subgroups of GL^(Z), for d ^ 6, and Aut(F £ j), for d ^ 5, are constructed as 
well. 



1. Introduction 

The goal of this paper is to prove the following result. 

Theorem 1.1. There exist automaton groups with unsolvable conjugacy problem. 

The question on solvability of the conjugacy problem was raised in 2000 for the 
class of self-similar groups generated by finite self-similar sets (automaton groups) 
by Grigorchuk, Nekrashevych and Sushchanskh [GNSQO] . Note that the word prob- 
lem is solvable for all groups in this class by a rather straightforward algorithm 
running in exponential time. Moreover, for an important subclass consisting of 
finitely generated, contracting groups the word problem is solvable in polynomial 
time. Given that our examples contain free nonabelian subgroups and that con- 
tracting groups, as well as the groups Pol(n), n > 0, do not contain such subgroups 
(see [NeklOj and [Sid04]), the following question remains open. 

Question 1.2. Is the conjugacy problem solvable in 

(i) all finitely generated, contracting, self-similar groups? 

(ii) the class of automaton groups in Pol(n), for n ^ 0? 

There are many positive results on the solvability of the conjugacy problem in 
automaton groups close to the first Grigorchuk group [Gri80| and the Gupta-Sidki 
examples GS83 . The conjugacy problem was solved for the first Grigorchuk group 
independently by Leonov |Leo98j and Rozhkov |Roz98j . and for the Gupta-Sidki 
examples by Wilson and Zaleskii |WZ97 . Grigorchuk and Wilson GW00] showed 
that the problem is solvable in all subgroups of finite index in the first Grigorchuk 
group. In fact, the results in [Leo98 , WZ971 IGW00] apply to certain classes of groups 
that include the well known examples we explicitly mentioned. In a recent work 
Bondarenko, Bondarenko, Sidki and Zapata BBSZ10 showed that the conjugacy 
problem is solvable in Pol(0). Lysenok, Myasnikov, and Ushakov provided the first, 
and so far the only, significant result on the complexity of the conjugacy problem in 
automaton groups by providing a polynomial time solution for the first Grigorchuk 
group |LMU10j . 
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The strategy for our proof of Theorem 11.11 is as follows. First, we observe the 
following consequence of a result by Bogopolski, Martino and Ventura IB MVlOj . 

Proposition 1.3. Let H be a finitely generated group, and T a finitely generated 
subgroup of Aut(iJ). IfT^ Aut(iJ) is orbit undecidable then H x T has unsolvable 
conjugacy problem. 

Since, for d ^ 4, examples of finitely generated orbit undecidable subgroups V 
in GLd(Z) are provided in BM VlOj , we obtain the existence of groups of the form 
Jj d x r with unsolvable conjugacy problem. Finally, using techniques of Brunner 
and Sidki [BS98|. we prove the following result, which implies Theorem ll.il 

Theorem 1.4. Let T be an arbitrary finitely generated subgroup of GLd(Z). Then, 
J/ 1 xi r is an automaton group. 

The examples of finitely generated orbit undecidable subgroups of GLd(Z), for 
d ^ 4 given in B MVlOj are based on Mikhailova's construction and are not finitely 
presented. By modifying the construction in BM VlOj . at the cost of increasing the 
dimension by 2, we determine finitely generated, orbit undecidable, free subgroups 
of GL d (Z), for d ^ 6. Note that, by |BMV10i Proposition 6.9.] and the Tits 
Alternative |Tit72] . every orbit undecidable subgroup T of GLd(Z) contains free 
nonabelian subgroups. By using the same technique (see Proposition 12. 2|) we also 
construct finitely generated, orbit undecidable, free subgroups of Aut(Fd), for d > 5, 
answering Question 6 raised in [BM VlOj . 

Proposition 1.5. (a) For d ^ 6, the group GLrf(Z) contains finitely generated, 
orbit undecidable, free subgroups. 

(b) For d ^ 5, the group Aut(Fd) contains finitely generated, orbit undecidable, 
free subgroups. 

This allows us to deduce the following strengthened version of Theorem 11.11 

Theorem 1.6. For every d ^ 6, there exists a finitely presented group G simulta- 
neously satisfying the following three conditions: 

i) G is an automaton group, 

ii) G is "L d -by-(f.g.-free) (in fact, G = Z d F m , with infective action <fi), 
Hi) G has unsolvable conjugacy problem. 

2. Orbit undecidability 
The main result in BMV10 can be stated in the following way. 

Theorem 2.1 (Bogopolski, Martino, Ventura |BMV10j ). Let G = H x F be a 

semidirect product (with F, H, and so G, finitely generated) such that 

(i) the conjugacy problem is solvable in F, 

(ii) for every f G F, (/) has finite index in the centralizer Ci?(/) and there is 
an algorithm that, given f , calculates coset representatives for (f) in Cp{f), 

(Hi) the twisted conjugacy problem is solvable in H. 
Then the following are equivalent: 

(a) the conjugacy problem in G is solvable, 

(b) the conjugacy problem in G restricted to H is solvable, 

(c) the action group {X g \ g £ G} ^ Aut(-ff) is orbit decidable, where X g denotes 
the right conjugation by g, restricted to H . □ 



THE CONJUGACY PROBLEM IN AUTOMATON GROUPS IS NOT SOLVABLE 3 



The conjugacy problem in G restricted to H asks if, given two elements u and 
v in H, there exists an element g in G such that u 9 — v. The orbit problem for 
a subgroup T of Aut(_ff ) asks if, given u and v in H , there is an automorphism 7 
in r such that 7(1/) is conjugate to v in H; we say that T is or&ii decidable (resp. 
undecidable) if the orbit problem for T is solvable (resp. unsolvable). Finally, the 
twisted conjugacy problem for a group H asks if, given an automorphism if G Aut(_ff ) 
and two elements u,v £ H, there is x G H such that v = ip{x)~ l ux. 

The implications (a) =>■ (6) (c) in Theorem 12. II are clear from the definitions, 
and do not require most of the hypotheses (as indicated in [BM VlOj . the only 
relevant implication is (c) =>■ (a))- Proposition ll.3i which is needed for our purposes, 
is an obvious corollary. 

Proposition 2.2. Let G be a group and H and K be subgroups of G such that 

(i) G=(H,K), 

(ii) the free group F 2 of rank 2 is a subgroup of Aut(K), 

(Hi) there exists a finitely generated orbit undecidable subgroup T ^ Aut(_ff), 

(iv) every pair of automorphisms a G Aut(_ff) and [3 G Aut(-fT) has a (necessarily 
unique) common extension to an automorphism of G, and 

(v) two elements of H are conjugate in G if and only if they are conjugate in H. 
Then, Aut(G) contains finitely generated, orbit undecidable, free subgroups. 

Proof. Let T = {gi, . . . , g m ) be an orbit undecidable subgroup of Aut(_ff) and F = 
(fi, . . . , f m ) be a free subgroup of rank m of Aut(A"). For, i = 1, ... ,rn, let s» be the 
common extension of Qi and ft to an automorphism of G and let T' = (si , . . . , s m ) ^ 
Aut(G). Since F is free of rank to, so is T'. 

Moreover, T' is orbit undecidable subgroup of Aut(G). Indeed, for u,v G H, 

(3 7 ' g r')(3t' g g) y(tt) = v*' ^=> (3 7 g r)(at' g g) = v*' <s=> 

<^ (37 G r)(3t G H) 7(u) = « f 

The second equivalence follows from (v), since 7(u),i> G H. The first comes from 
the construction, since, for every group word w(x±, . . . , x m ), the automorphisms 
7' = w(si, . . . , s m ) G r' and 7 = w(gi, . . . ,g m ) G T agree on i/. Therefore, the 
orbit problem for the instance u, v G H with respect to V ^ Aut(H) is equivalent 
to the orbit problem for the instance u,v G H ^ G with respect to T' ^ Aut(G), 
showing that an algorithm that would solve the orbit problem for V could be used 
to solve the orbit problem for T as well. Thus V is orbit undecidable. □ 

Proof of Proposttion[Tj[ (a) For d > 6, let G = Z d , H = Z d ~ 2 , K = 1? , and 
G = H ffi K . All requirements of Proposition 12.21 are satisfied. In particular, (hi) 
holds by |BMV10l Proposition 7.5], and (v) holds since conjugacy is the same as 
equality in both H and G. 

(b) For d > 5, let G = F d , H = F d - 2 , K = F 2 , and G = H * K. All re- 
quirements of Proposition 12.21 are satisfied. In particular, (hi) holds by |BMV10( 
Subsection 7.2], and (v) holds since the free factor H is malnormal in G. □ 

3. Self-similar groups and automaton groups 

Let X be a finite alphabet on k letters. The set X* of words over X has the 
structure of a rooted A;-ary tree in which the empty word is the root and each vertex 
u has k children, namely the vertices ux, for x in X. Every tree automorphism 
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fixes the root and permutes the words of the same length (constituting the levels 
of the rooted tree) while preserving the tree structure. 

Let g be a tree automorphism. The action of g on X* can be decomposed 
as follows. There is a permutation 7r ff of X, called the root permutation of g, 
determined by the permutation that g induces on the subtrees below the root (the 
action of g on the first letter in every word), and tree automorphisms g\ x , for x 
in X, called the sections of <?, determined by the action of g within these subtrees 
(the action of g on the rest of the word behind the first letter). Both the root 
permutation and the sections are uniquely determined by the equality 

(!) g{xw) = 7T g (x)g\ x (w), 

for x in X and w in X*. 

A group or a set of tree automorphisms is self-similar if it contains all sections 
of all of its elements. A finite automaton is a finite self-similar set. A group G(A) 
of tree automorphisms generated by a finite self-similar set A is itself self-similar 
and it is called an automaton group (realized or generated by the automaton A). 
The elements of the automaton are often referred to as states of the automaton and 
the automaton is said to operate on the alphabet X. 

The boundary of the tree X* is the set X" of right infinite words x\Xix% 
The tree structure induces a metric on X" inducing the Cantor set topology. The 
metric is given by d(u,v) = 2l „ 1 Al , | , for u ^ v, where \u A v\ denotes the length of 
the longest common prefix of u and v. The group of isometries of the boundary 
X M and the group of tree automorphism of X* are canonically isomorphic. Every 
isometry induces a tree automorphism by restricting the action on finite prefixes, 
and every tree automorphism induces an isometry on the boundary through an 
obvious limiting process. The decomposition formula ([l) for the action of tree 
automorphisms is valid for boundary isometries as well (w is any right infinite word 
in this case). 

4. Automaton groups with unsolvable conjugacy problem 

Let M = {Mi, . . . ,M m } be a set of integer d x d matrices with non-zero de- 
terminants. Let n ^ 2 be relatively prime to all of these determinants (thus, each 
Mi is invertible over the ring Z n of n-adic integers. For an integer matrix M 
and an arbitrary vector v with integer coordinates, consider the invertible affinc 
transformation M v : Z^ — »■ ZjJ given by M v (u) = v + Mu, and let 

G M ,n = ({Mv I M G M, v g Z d }> 

be the subgroup of AfLj(Z n ) generated by all the transformations of the form M v , 
for M € M and v G Z d . Denote by r v the translation Z„ — >• Z„, u h-> v + u, and 
by ej the z-th standard basis vector. Since M v = t v Mq, we have 

(2) G M , n = {{Mo \ MeM}U{r ei \i = l,...,d}) < Aff d (Z n ). 

Lemma 4.1. // all matrices in Ai are invertible over Z, then Gm,u — Z d x T, 
where T — {M) ^ GLrf(Z); in particular, Gm,u does not depend on n. 

Proof. If M is an invertible matrix over Z, and v G Z d , then M v G ASd(Z n ) restricts 
to a bijective affine transformation M v G Affd(Z). Hence, we can view Gm,u as a 
subgroup of Aff £ ;(Z) and, in particular, it is independent from n; let us denote it 
by G M - 
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Clearly, the subgroup of translations T = (r ei , • • ■ , r e<J ) of Gm is free abelian of 
rank d, T ~ Z d . Since each of the transformations Mo, for M £ Ai, acts on Z d by 
multiplication by M, the subgroup (Mo | M £ Ai) of Gm is isomorphic to V and 
may be safely identified with it. The subgroups T and T intersect trivially, since 
every nontrivial element of T moves the zero vector in Z d , while no element of T 
does. For M £ Ai U Mr 1 (where Ai~ x is the set of integer matrices inverse to the 
matrices in Ai), j = 1, . . . , d, and u £ Z d , 

M Te 3 (Afo) _1 (u) = M r e , (Af _1 u) = M (e j + M" x u) = Mej + u = 

Jlllj m 2,j m d.j I \ 

— T ei T e2 ' • • • T e<i (Uj, 

where mjj is the (i, j)-entry of M. Therefore, for M £ U .M _1 and j = 1, . . . , d, 

(3) More, (Mo)" 1 = r™(^>r™( 2 ^ ■ ■ ■ . 

It follows that the subgroup T = Z d is normal in and = Z d x T. □ 

Remark 4.2. The equality ((3|) is correct (over Z n ) for any integer matrix with 
non-zero determinant relatively prime to n. When Ai = {M} consists of a single 
d x d integer matrix M = (rriij) of infinite order and determinant k ^ relatively 
prime to n, the multiplication by M embeds Z d into an index \k\ subgroup o f Z d an d 
Gm,u is the ascending HNN extension of Z d by a single stable letter (see |BS06 ), 
i.e., 

G M ,n — ( ai,...,a d ,t | [oi,o 3 -] = 1, to^ -1 = a" ll,J •••a™ d ' J , for 1 < i,j s^d). 

The goal now is to show that the groups Gm,u constructed in this way, can all 
be realized by finite automata and so, they are automaton groups. 

The elements of the ring Z„ may be (uniquely) represented as right infinite words 
over the alphabet Y n — {0, . . . , n — 1}, through the correspondence 

2/iZ/2?/3 ■ ■ • < — ► V\ + 2/2 • n + y 3 ■ n 2 H , 

while the elements of the free d-dimensional module Z^, viewed as column vectors, 
may be (uniquely) represented as right infinite words over the alphabet X n = Y„ — 
{(t/i, . . . , yd) T Hi £ Yn, i — 1) • • • ) d} consisting of column vectors with entries in 
Y n . Note that \Y n \ — n and \X n \ = n d . 

For a vector v with integer coordinates define Mod(v) and Div(v) to be the vec- 
tors whose coordinates are the remainders and the quotients, respectively, obtained 
by dividing the coordinates of v by n, i.e., the unique integer vectors satisfying 
v = Mod(v) + nDiv(v), with Mod(v) £ X n . 

Lemma 4.3. For every vector v with integer coordinates, and every element X1X2X3 . . . 
in the free module Z^ (where Xi, X2, X3, . . . are symbols in X n ), 

(4) M v (xxx 2 x 3 • • • ) = Mod(v + Mxi) + nM£)j w(v+Mxi) (x 2 x 3 X4 • ■ • ). 
Proof. Indeed, 

M v (xix 2 x 3 • • • ) = v + Mxix 2 x 3 • • • = v + M(xi + n(x 2 x 3 x 4 • • • )) 
= v + Mxi + ?iA/x 2 x 3 X4 • • • 

= Mod(v + Mxi) + nDiv(v + Mxi) + nMx 2 x 3 x 4 • • • 
= Mod(v + Mxi) + n(Div(v + Mx x ) + Mx 2 x 3 x 4 • • • ) 
= Mod(v + Mxi) + nM Div(v+Mxi) (x 2 x 3 X4 • • • )• 
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□ 

Let \ \M\ | be the maximal absolute row sum norm of M, i.e. 1 1 M \ \ = max^ Ylj=i \ m i,j I > 
where rrtij is the (i, j')-entry of M. Define Vm to be the finite set of integer vectors 
v for which each coordinate is between — ||M|| and ||M|| — 1, inclusive. Note that 
Vm is finite and contains (2||M||) d vectors. 

Definition 4.4. For an integer matrix M, define an automaton Am,u operating 
on the alphabet X n as follows: the set of states is Sm,ti — { m v | v G Vm}, and the 
root permutations and the sections are, for x in X n , defined by 

(5) m v (x) = Mod(v + Mx) and m v | x = ^Div(v+A/x)- 

The automaton Am,u is well defined (it is easy to show that, for v 6 Vm and 
x G X n , the entries of the vector v + Mx are bounded between — ||M||n and 
\\M\\n - 1, and hence Div(v + Mx) G Vm)- 

Lemma 4.5. For every state m v of the automaton Am.u, o,nd every element u = 
X1X2X3 • • • of the free module Z„ (i.e. every right infinite word over X n ). 

m v (u) = M v (u). 

Proof. Follows directly from the definition of the root permutations and the sections 
of m v in §5§ and equality (jlj) describing the action of M v . □ 

Definition 4.6. Let Am.u be the automaton operating on the alphabet X n and 
having 2 d YliLi states obtained by taking the (disjoint) union of the au- 

tomata AMx,n, ■ ■ ■ , AM m ,n- 

Proposition 4.7. The group Gm.u can be realized by a finite automaton acting on 
an alphabet of size n and having no more 

than2 d Y,T=i \ \ M i\\ d states, where \ \Mi\\ 
is the maximum absolute row sum norm of Mi, for i — 1, . . . , m. 

Proof. The automaton Am,u satisfies the required conditions, and generates pre- 
cisely the group Gm.u- This follows directly from ((2} and Lemma [43] once it 
is observed that Am,u has enough states to generate Gm.u- However, this is 
clear, since each of the automata Am.u, for M G A4, has at least d + 1 states, 
m , m_ei, ■ • ■ ,«i-e d , and mo^-e.,) -1 = r ej , for j = 1, . . . , d. □ 

Theorem 11.41 is an immediate corollary of Lemma 14.11 (ii) and Proposition 14.71 

Proof of Theorem ] l.b\ Let d ^ 6 and let F be an orbit undecidable, free subgroup of 
rank m of GLd(Z) (such a group exists by Proposition ll.5[) . Let M. = {Mi, . . . , M m } 
be a set of invertible integer d x d matrices generating F = (M). Fix n ^ 2 
and consider the group G = Gm.u- By Proposition 14.71 G is generated by the 
finite automaton Am,ti, so it is an automaton group. By Lemma 14. II (ii). G does 
not depend on n and is in fact isomorphic to % d x F (since all matrices in M 
are invertible over Z); so, it is a Z d -by-free group. Finally, by Proposition ll.3[ 
G = Gm.ii has unsolvable conjugacy problem. □ 

Theorem 11.11 is an immediate corollary of Theorem 11.61 
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